A storage system is a computer that provides storage service relating to the organization of information on writeable persistent storage devices, such as memories, tapes or disks. The storage system is commonly deployed within a storage area network (SAN) or a network attached storage (NAS) environment. When used within a NAS environment, the storage system may be embodied as a file server including an operating system that implements a file system to logically organize the information as a hierarchical structure of directories and files on, e.g. the disks. Each “on-disk” file may be implemented as a set of data structures, e.g., disk blocks, configured to store information, such as the actual data for the file. A directory, on the other hand, may be implemented as a specially formatted file in which information about other files and directories are stored.
The file server, or filer, may be further configured to operate according to a client/server model of information delivery to thereby allow many client systems (clients) to access shared resources, such as files, stored on the filer. Sharing of files is a hallmark of a NAS system, which is enabled because of semantic level of access to files and file systems. Storage of information on a NAS system is typically deployed over a computer network comprising a geographically distributed collection of interconnected communication links, such as Ethernet, that allow clients to remotely access the information (files) on the file server. The clients typically communicate with the filer by exchanging discrete frames or packets of data according to pre-defined protocols, such as the Transmission Control Protocol/Internet Protocol (TCP/IP).
In the client/server model, the client may comprise an application executing on a computer that “connects” to the filer over a computer network, such as a point-to-point link, shared local area network, wide area network or virtual private network implemented over a public network, such as the Internet. NAS systems generally utilize file-based access protocols; therefore, each client may request the services of the filer by issuing file system protocol messages (in the form of packets) to the file system over the network. By supporting a plurality of file system protocols, such as the conventional Common Internet File System (CIFS), the Network File System (NFS) and the Direct Access File System (DAFS) protocols, the utility of the filer may be enhanced for networking clients.
A SAN is a high-speed network that enables establishment of direct connections between a storage system and its storage devices. The SAN may thus be viewed as an extension to a storage bus and, as such, an operating system of the storage system enables access to stored information using block-based access protocols over the “extended bus”. In this context, the extended bus is typically embodied as Fibre Channel (FC) or Ethernet media adapted to operate with block access protocols, such as Small Computer Systems Interface (SCSI) protocol encapsulation over FC (FCP) or TCP/IP/Ethernet (iSCSI). A SAN arrangement or deployment allows decoupling of storage from the storage system, such as an application server, and some level of storage sharing at the application server level. There are, however, environments wherein a SAN is dedicated to a single server.
It is advantageous for the services and data provided by a storage system to be available for access to the greatest degree possible. Accordingly, some computer systems provide a plurality of storage systems organized in a cluster, with a property that when a first storage system fails, a second storage system is available to take over and provide the services and the data otherwise served by the first storage system. In particular, when the first storage system fails, the second storage system in the cluster assumes the tasks of processing and handling any data access requests normally processed by the first storage system. One such example of a cluster configuration is described in U.S. patent application Ser. No. 09/625,234 entitled NEGOTIATING TAKEOVER IN HIGH AVAILABILITY CLUSTER by Samuel M. Cramer, et al., the contents of which are hereby incorporated by reference.
In a typical cluster environment, there may be several processes executing on each storage system (“cluster partner”) that desire to communicate with corresponding “peer” processes executing on the other storage system partner in the cluster. One exemplary process is a cluster failover (CFO) monitoring process that determines if a cluster partner has failed and whether a takeover operation of the storage (e.g., disks) served by the failed storage system should be initiated. To that end, the CFO monitoring process sends routine “heartbeat” signals to its corresponding CFO monitoring process to alert the cluster partner that its other partner is operating without any serious errors that would necessitate a failover.
Typically, each of these peer processes utilizes its own protocol implementation for opening, closing, and managing network data connections to its corresponding peer process. However, this results in inefficient duplication of protocol or communication management routines. In addition, such duplication may increase the difficulty of coordinating information between peer processes on cluster partners in the event of loss of a communication medium (e.g., a cluster interconnect) coupling the partners. Specifically, in the event of a failure of the cluster interconnect, the various processes lose their capability to provide peer-to-peer communication with their respective cluster partner peer processes. This lack of communication adversely affects the cluster by preventing the cluster partners from coordinating state and other configuration information between them. For example, data loss may occur as synchronization with respect to a non-volatile random access memory (NVRAM) shadowing process executing on the partner is disrupted.
As noted each peer process typically creates and manages its own peer connection with its corresponding peer process on a cluster partner. The handshaking and capabilities exchange among processes needed to create and manage the peer connection are performed in accordance with a conventional protocol implementation, such as the Virtual Interface (VI) protocol. The VI protocol is typically implemented by a VI layer of a storage operating system executing on each storage system of the cluster. However, a peer process may not begin communicating with its corresponding peer process on the cluster partner until the VI layer has been loaded during a boot sequence of the storage system, which may consume a substantial amount of time. By requiring a peer process, such as the CFO monitoring process, to wait until the VI layer of the storage operating system has been loaded and executed, significant amounts of time may pass and unnecessary failover operations may occur, especially when a storage system suffers a minor transient error condition. Cluster performance requires peer processes to be in communication with their corresponding peer process on the cluster partner as soon as possible during the boot sequence.
Another disadvantage of a conventional cluster environment is the inability to balance communication “loads” among peer processes executing on the cluster partner. In the conventional cluster environment, all peer-to-peer communications typically occur over a single cluster interconnect. Certain peer processes may consume inordinate amounts of bandwidth available over a given cluster interconnection, thereby reducing the bandwidth available for other peer processes. For example, the NVRAM shadowing process may, during periods of heavy loads, consume a substantial amount of the cluster interconnect bandwidth. The present invention is directed, in part, to providing a technique for balancing loads transferred between processes of a cluster environment.